Online Non-Parametric Regression

We establish optimal rates for online regression for arbitrary classes of regression functions in terms of thesequential entropy introduced in [14]. The optimal rates are shown to exhibit a phase transition analogous to thei.i.d./statistical learning case, studied in [16]. In the frequently encountered situation when sequential entropyand i.i.d. empirical entropy match, our results point to the interesting phenomenon that the rates for statisticallearning with squared loss and online nonparametric regression are the same.In addition to a non-algorithmic study of minimax regret, we exhibit a generic forecaster that enjoys the es-tablished optimal rates. We also provide a recipe for designing online regression algorithms that can be computa-tionally efficient. We illustrate the techniques by deriving existing andnew forecasters for the case of finite expertsand for online linear regression.